System  of  Systems  (SoS)  M&S 
VV&A  Decomposition: 


Integrated  System  Level  VV&A  (I SLA) 


Dr.  Jeffrey  S.  Strickland 
Certified  Modeling  &  Simulation  Professional 
MDA  M&S  V&V  Technical  Director 


Approved  for  Public  Release 
08-MDA-4059  (5  JAN  09) 


Distribution  Statement  A: 

Approved  for  public  release;  distribution  is  unlimited 


1 


Report  Documentation  Page 


Form  Approved 
0MB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 


1.  REPORT  DATE 

JAN  2009 


2.  REPORT  TYPE 


3.  DATES  COVERED 

00-00-2009  to  00-00-2009 


4.  TITLE  AND  SUBTITLE 

System  of  Systems  (SoS)  M&S  VV&A  Decomposition:  Integrated  System 
Level  VV&A  (ISLA) 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 


5c.  PROGRAM  ELEMENT  NUMBER 


6.  AUTHOR(S) 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


7.  PEREORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Missile  Defense  Agency ,7100  Defense 
Pentagon, Washington, DC, 20301-7100 


8.  PEREORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 


11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 


13.  SUPPLEMENTARY  NOTES 

Live- Virtual  Constructive  Conference,  12-15  Jan  2009,  El  Paso,  TX 


14.  ABSTRACT 


15.  SUBJECT  TERMS 


17.  LIMITATION  OE 

18.  NUMBER 

ABSTRACT 

OE  PAGES 

Same  as 

18 

Report  (SAR) 

16.  SECURITY  CLASSIEICATION  OE: 


a.  REPORT 

unclassified 


b.  ABSTRACT 

unclassified 


c.  THIS  PAGE 

unclassified 


19a.  NAME  OE 
RESPONSIBLE  PERSON 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Purpose 


•  The  purpose  of  this  brief  to  propose  an 
approach  for  performing  “system  of 
system”  level  V&V  of  the  BMDS 
representations. 


Approved  for  Public  Release 
08-MDA-4059  (5  JAN  09) 


2 


BMDS  SoS  Hierarchical  View 


System-Level  (SoS)  V&V  Element-Level  (System)  V&V 

•  SoS  Interfaces  •  Enterprise  Rules  •  System  Interfaces  •  Data  model 

•  SoS  Behavior  •  Scenarios  •  System  Rules 

•  SoS  Framework  •  Metadata  model  •  System  Framework 

•  Boundaries 
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Lifecycle 

Applications 

Provides  for  common 
simulation  management 


Optimistic 

Processing 


Offline 

Accessible 


BMDS  SoS  M&S  W&A  Tool  Metric  Dendritic 


MOE  =  Measure  of  Effectiveness 
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TPM  =  Technicai  Performance  Measure  (some  my  be  “critical”,  CPM) 
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BMDS  SoS  M&S  W&A  Event  Metric  Dendritic 


CAi  =  Criticai  Analysis  Issue 
MOP  =  Measure  of  Performance 

TPM  =  Technical  Performance  Measure  (some  my  be  “critical”,  CPM) 


AC  =  Acceptability  Criteria 
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Test  Objective  Relationship  to  Acceptability  Criteria 
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Issues 


•  Which  interfaces  are  “truly”  SoS  level 

•  How  do  we  V&V  interfaces? 

•  What  in  DSA  corresponds  to  a 
Federation  Object  Model  (FOM)  in 
HLA? 

•  How  do  we  achieve  meta-model 
validation  in  the  absence  of  a 
conceptual  model? 

•  What  constitutes  a  metadata  model? 
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SoS  M&S  Verification  &  Vaiidation  Requirements 
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Testing  the  Theory  Through  System-Level  Post- 

Flight  Reconstruction  (U) 


-  System-Level  Post-Flight  Reconstruction  (PFR): 

»  Manually  recreate  and  run  a  past  flight  test  scenario  in  a  test 
venue  performing  system-level  comparative  analysis  of  the  real- 
world  performance  to  the  output  of  the  test  venue  assessing  the 
results  and  determining  if  system-level  anomalies  exist  in  the 
M&S 


-  System-Level  Anchoring  (SLA): 

»  Perform  root  cause  analysis  of  the  system-level  anomalies  found 
in  the  PFR;  generate,  test  and  implement  M&S  Improvements  to 
address  anomalies 
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M&S  System-Level  PFR  Preparation  Process  (U) 


TC,  DEET, 
MDSE  (GT), 
Integration 
Team  (PA) 
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Terms  of  Reference  (1) 


•  System:  A  combination  of  interacting  elements  organized  to  achieve  one  or 
more  stated  purposes. 

•  System  Element:  a  member  of  a  set  of  elements  that  constitute  a  system.  A 
system  element  is  a  discrete  part  of  a  system  that  can  be  implemented  to 
fulfill  a  specified  requirement. 

•  System  of  Systems:  A  combination  of  interacting  functional  system  elements, 
which  are  themselves  systems,  organized  to  achieve  a  stated  operational 
capability. 

•  Interface:  At  point  at  which  independent  systems  or  diverse  groups  interact. 
The  devise  or  system  by  which  interaction  at  an  interface  is  effected. 

•  Enterprise  Rules:  Rules  that  govern  the  interaction  of  system  elements; 
serve  the  role  of  “local  government”  within  the  SoS;  specified  in  the 
Simulation  Federation  Object  Model  (framework),  for  example. 

•  Verification:  The  process  of  determining  that  a  model  implementation  and  its 
associated  data  accurately  represent  the  developer's  conceptual  description 
and  specifications. 

•  Validation:  The  process  of  determining  the  degree  to  which  a  model  and  its 
associated  data  are  an  accurate  representation  of  the  real  world  from  the 
perspective  of  the  intended  uses  of  the  model. 
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Terms  of  Reference  (2) 


•  Measure  of  Effectiveness  (MOE):  A  qualitative  or  quantitative  measure  of 
aggregate  performance  or  a  characteristic  of  a  model,  simulation  or  system 
that  indicates  the  degree  to  which  it  performs  the  task  or  meets  an 
operational  objective  or  requirement  under  specified  conditions. 

•  Measure  of  Performance  (MOP):  The  measure  of  how  the  system/individual 
performs  its  functions  in  a  given  environment  (e.g.,  number  of  targets 
detected,  reaction  time,  number  of  targets  nominated,  susceptibility  of 
deception,  task  completion  time).  It  is  closely  related  to  inherent  parameters 
(physical  and  structural)  but  measures  attributes  of  system  behavior. 

•  Metadata:  Information  describing  the  characteristics  of  data;  data  or 
information  about  the  meaning  of  data;  descriptive  information  about  an 
organization's  data,  data  activities,  systems,  and  holdings. 

•  Metamodel:  A  model  of  a  model.  Metamodels  are  abstractions  of  the  M&S 
being  developed,  which  use  functional  decomposition  to  show  relationships, 
paths  of  data  and  algorithms,  ordering,  and  interactions  between  model 
components  and  subcomponents.  Metamodels  allow  the  software  engineers 
who  are  developing  the  model  to  abstract  details  to  a  level  that  subject 
matter  experts  can  validate. 

•  Modeling  and  Simulation  (M&S):  The  use  of  models  and  simulations,  either 
statically  or  over  time,  to  develop  data  as  a  basis  for  making  managerial  or 
technical  decisions.  This  includes  but  is  not  limited  to,  emulators,  prototypes, 
simulators,  and  stimulators. 
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System-Level  V&V  -  Roles  &  Responsibilities 
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How  Integrated  V&V  Supports  Accreditation 
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Small  (laptop)  Mod.  (rack)  Large  (room) 


Simulation  Use  Cases 


Use  Case  Clusters 
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•  BMD  Simulation  Use  Cases 

-Tend  to  cluster  when  organized 
by  support  personnel  and 
hardware  footprint 

-A  single  simulation  (conceptual 
model)  is  likely  to  poorly  serve 
the  complete  space  of  use  cases 


-Addressed  by  engineering 
simulation  models  integrated 
through  EMF 

-  Marginal  improvement  over 
today’s  (PA07)  executions 

-  Ideally  suited  for  interaction 
with  HWIL  simulation  and 
ground  and  flight  tests 

•  Medium  Fidelity  Cases 

-  Necessary  for  support  of 
concept  exploration,  future 
capabilities,  high  run  count 
statistical  analyses 

-Gaining  credibility  will  require 
sound  modeling,  solid 
benchmarking  to  high  fidelity 
simulations,  and  repeated  use 
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SISO  Fidelity  ISG  Concepts 
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Fidelity  is  defined  as  the  accuracy  of  the  representation  when  compared 
to  the  real-world  [DoD  5000.59-M;  DoD  M&S  Glossary,  Dec  1997]. 
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